Skip to content

Drop the per-user and cliff columns from v2 report rows - #4335

Draft
anth-volk wants to merge 10 commits into
v2plan/c1-40a-stop-mapping-report-columnsfrom
v2plan/c1-41-drop-report-columns
Draft

anth-volk wants to merge 10 commits into
v2plan/c1-40a-stop-mapping-report-columnsfrom
v2plan/c1-41-drop-report-columns

Conversation

@anth-volk

Copy link
Copy Markdown
Collaborator

Fixes #4334

Summary

Adds v2 revision 4e30076d4b9f, on bc334f9877b7. It drops:

Table Columns Also dropped
reports label, description, user_id fk_reports_user_id_users, ix_reports_user_id
report_runs trigger, baseline_cliff_gap, baseline_cliff_share, reform_cliff_gap, reform_cliff_share ck_report_runs_cliff_impact_complete
budget_window_runs trigger
  • The Modal worker's declared UPDATE on the four cliff columns goes.
  • Account deletion no longer touches reports.
  • v2_report_run_trigger stays, for user_report_run_associations. budget_window_runs.recalculate stays.

The drops can't be undone by a downgrade

The downgrade adds the columns, key, index and check back with every value null. Nothing copies the values first, because these rows name no user to copy them to.

Before merging, an operator exports the stored values from staging and from production, and keeps the result outside this repository, because labels and descriptions are users' text. These queries only read:

SELECT id, created_at, label, description, user_id FROM reports
WHERE label IS NOT NULL OR description IS NOT NULL OR user_id IS NOT NULL;
SELECT id, report_id, created_at, trigger, baseline_cliff_gap,
       baseline_cliff_share, reform_cliff_gap, reform_cliff_share
FROM report_runs
WHERE trigger IS NOT NULL OR baseline_cliff_gap IS NOT NULL;
SELECT id, report_id, created_at, trigger FROM budget_window_runs
WHERE trigger IS NOT NULL;

What must be in production before this merges

The release workflow upgrades the v2 schema before the new Cloud Run revision takes traffic, so the release already serving runs on the dropped schema.

  1. API v1's Store one cliff result per simulation run in the v2 database, with grants #4327, Read report cliffs from simulation runs' cliff results, and take include_cliffs on re-runs #4329 and Stop mapping the report columns the next v2 revision drops #4333. Together they stop API v1 naming any of these columns; without them every /v2/reports read fails during the deploy.
  2. The simulation service's redone Calculate each simulation run's cliff gap and share as its own cliff result policyengine-sim-api#1096 (C1 · 38), as its default worker's v2 app. Its v2 runtime on main still writes the four cliff columns, so without it every economy report run's completion fails.

C1 · 42 follows: the simulation service's table models lose these columns and its pinned API v1 commit moves to this merge. Until then its schema comparison fails against this schema; requests are not affected.

Migration notes

  • Route group affected: /v2/reports. This PR changes no HTTP response; Stop mapping the report columns the next v2 revision drops #4333 removed trigger.
  • One post-generation correction, commented in the revision: the downgrade's check condition gets back the first and last parenthesis autogenerate strips.
  • .github/scripts/test_alembic_v2_lifecycle.sh gains the new test module. It reads no new environment variable: V2_MIGRATION_DATABASE_URL, V2_ALEMBIC_DISPOSABLE_TEST and UV_PYTHON are literal values in alembic-v2-check.yml's job env.

Testing

  • The tests this PR adds or changes pass by node ID against disposable PostgreSQL 17, including upgrade, downgrade against a parent schema built from the base, and the privilege steps from the Modal worker's old cliff grants.
  • alembic -c alembic-v2.ini check finds nothing to generate, and reconcile_v2_privileges.py apply and verify pass.
  • The simulation service's redone C1 · 38 code ran 205 database tests by node ID against this schema. Every failure came from test seeds or test assertions naming a dropped column, and none from its own statements. Its /ready privilege check finds nothing missing.
  • ruff, ruff format, lint-imports, python -m scripts.rules validate, run_quality_guards.py and make typecheck-v2 pass. Explicit mypy on the changed policyengine_api/data/v2/ files adds no error to the base branch's.
  • The full suites were not run locally.

🤖 Generated with Claude Code

anth-volk and others added 6 commits November 19, 2026 02:23
The per-user columns of reports, report runs and budget-window runs, and
the report runs' cliff columns, go in a new revision. Report reads stop
answering a run's trigger, the privilege manifest stops naming the cliff
columns, and account deletion needs no reports.user_id. Rows the tests
store at the head revision stop naming the columns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Revision 4e30076d4b9f drops reports.label, description and user_id,
report_runs.trigger and its four cliff columns, and
budget_window_runs.trigger, with the foreign key, index and check that
depend on them. The downgrade adds them back empty. /v2/reports run items
no longer have a trigger, the Modal worker's grants no longer name the
cliff columns, and account deletion no longer touches reports.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…c1-41-drop-report-columns

# Conflicts:
#	policyengine_api/data/v2/models/reports.py
#	policyengine_api/data/v2/models/users.py
#	tests/integration/test_v2_report_cliff_reads.py
The revision's tests compare it and its downgrade with the parent schema
built from the base, and CI's migration lifecycle job runs them. A test
assertion that could not fail goes, and the privilege docstring and the
migration guide say what must happen before merging and after a downgrade.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The revision gets its generation command and a description, the privilege
test starts from the Modal worker's old cliff grants, the workflow test
pins the new lifecycle module, and the downgrade guides say that going
below the shared reports revision needs no reports or runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.62%. Comparing base (6fcb6b2) to head (bd6208e).

Additional details and impacted files
@@                            Coverage Diff                             @@
##           v2plan/c1-40a-stop-mapping-report-columns    #4335   +/-   ##
==========================================================================
  Coverage                                      93.61%   93.62%           
==========================================================================
  Files                                            409      409           
  Lines                                          23352    23349    -3     
  Branches                                        3656     3656           
==========================================================================
- Hits                                           21862    21861    -1     
+ Misses                                           875      874    -1     
+ Partials                                         615      614    -1     
Flag Coverage Δ
integration 57.92% <100.00%> (-0.01%) ⬇️
unit 90.92% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

The reports cases of test_a_malformed_or_incomplete_key_is_refused still
named label, which revision 4e30076d4b9f drops, so they failed on the
missing column instead of the key check. Each case now also names the
check constraint it expects, so a refusal by a different check fails it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…c1-41-drop-report-columns

# Conflicts:
#	docs/engineering/skills/alembic-migrations.md
anth-volk and others added 2 commits November 20, 2026 01:16
…c1-41-drop-report-columns

# Conflicts:
#	docs/migration/stage-12a-user-login.md
#	docs/migration/stage-9-v2-metadata.md
#	tests/unit/v2/test_alembic_v2_revisions.py
Merging the dynamics revision through C1 · 40a brought back the cliff
impacts test's head assertion, which this branch had removed: revision
4e30076d4b9f is the head, and the chain test already asserts that.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch was previously deployed

1 inactive deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant